An exit-time approach to e-entropy 
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An efficient approach to the calculation of the e-entropy is proposed. The method is based on the 
idea of looking at the information content of a string of data, by analyzing the signal only at the 
instants when the fluctuations are larger than a certain threshold e, i.e., by looking at the exit-time 
statistics. The practical and theoretical advantages of our method with respect to the usual one are 
shown by the examples of a deterministic map and a self-affine stochastic process. 
PACS: 05.45.-a, 05.45.Tp 



The problem of quantifying the degree of complexity of 
an evolving system is ubiquitous in natural science (for 
a nice review see |J). The typical questions may range 
from the aim to distinguish between stochastic or chaotic 
systems to the more pragmatic goal of determining the 
degree of complexity (read predictability) at varying the 
resolution in phase-space and time 

From the pioneering works of Shannon Q and 
Kolmogorov ||, a proper mathematical tool, the 
Kolmogorov-Sinai {KS) entropy, hxs, has been devel- 
oped to address the above question quantitatively and 
unambiguously (in principle). The main idea is very nat- 
ural: we must look at the information contained in the 
time evolution as a probe of the underlying dynamics. 
This is realized by studying the symbolic dynamics ob- 
tained by assigning different symbols to different cells of 
a finite partition of the phase space. The probability dis- 
tribution of allowed sequences (words) is determined by 
the dynamical evolution. The average information-gain 
is defined by comparing sequences of length to and to + 1 , 
in the limit of large m. Letting the length of the words, 
to, to infinity and going to infinitely refined partition, 
one obtains the ifS'-entropy, which is a measure of the 
degree of complexity of the trajectory. The ifS'-entropy 
determines also the rate of information transmission nec- 
essary to unambiguously reconstruct the signal. 
Unfortunately, only in simple, low dimensional, dynami- 
cal systems such a procedure can be properly carried out 
with conventional methods J||||]]. The reason is that 
for high dimensional systems the computational resources 
are not sufficient to cope with the very high resolution 
and extremely long time series required. Moreover, in 
many systems, like in turbulence, the existence of non- 
trivial fluctuations on different time and spatial scales 
cannot be captured by the ifS'-entropy. This calls for a 
more general tool to quantify the degree of predictabil- 
ity which depends on the analyzed range of scales and 
frequencies. This was the aim leading Shannon M and 
Kolmogorov ^ to introduce the so-called e-entropy, later 
generalized to the (e, r)-entropy |§J|. Conceptually it 
corresponds to the rate of information transmission nec- 



essary to reconstruct a signal with a finite accuracy e, 
and with a sampling time interval r. The naive (e, r)- 
computation is usually performed by looking at the Shan- 
non entropy of the coarse grained dynamics on an (e, r)- 
grid in the phase-space and time. This method suffers of 
so many computational drawbacks that it is almost use- 
less for many realistic time-series . Another attempt in 
this direction is the introduction of the Finite Size Lya- 
punov Exponent Q. 

The aim of this letter is to introduce an alternative ap- 
proach for the determination of the (e, r)-entropy, based 
on the analysis of cxit-timcs. In a few words, the idea 
consists in looking at the information-content of a string 
of data, without analyzing the signal at any fixed time, 
t, but only when the fluctuations are larger than some 
fixed threshold, e. This simple observation allows for a 
remarkable improvement of the computational possibil- 
ity to measure the (e, r)-entropy as will be discussed in 
detail later. 

We believe that the approach presented hereafter is 
unavoidable in all those cases when either the high- 
dimensionality of the underlying phase space, or the ne- 
cessity to disentangle non-trivial correlations at different 
analyzed time scales, leads to the failure of the standard 
methods. 

Let us just briefly recall the conventional way to calcu- 
late the (e, r)-entropy for the case of a time-continuous 
signal x(t) G IR, recorded during a (long) time inter- 
val T. One defines an e-grid on the phase-space and 
a r-grid on time. If the motion is bounded, the tra- 
jectory visits only a finite number of cells; therefore to 
each subsequence of length n ■ r from x{t) one can as- 
sociate a word of length n, out of a finite alphabet: 
(e,r) = (St,S t+T , S , t+ („_i) T ), where S t labels the 
cell containing x(t). From the probability distribution 
of the above words one calculates the block entropies 
H n (e, T ): 

H n (e,r) = - ]T P(W n (e,r))lnP(W n {e,r)). (1) 
the (e, r)-entropy per unit time, /i(e, r) is finally defined 
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as: 



h n (e,r) = -[H n+1 (e, T ) - H n (e, T )} , (2) 

T 

h(e,r) = lim h n (e,r) = - lim -ff„(e,r) , (3) 

n — >oo T n — >oo Tl 

where for practical reasons the dependence on the de- 
tails of the partition is ignored, while the rigorous def- 
inition is given in terms of the infimum over all possi- 
ble partitions with elements of diameter smaller than e 
P). Notice that the above defined (e, r)-entropy is noth- 
ing but the Shannon-entropy of the sequence of sym- 
bols (S t , S t +T, ■ ■ ■) associated with the given signal. The 
Kolmogorov-Sinai entropy, hus, is obtained by taking 
the limit (e, t) — » 0: 



h K s = 



lim lim hit. t) 



(4) 



In the case of discrete-time systems, one can define h(e) = 
h(e,T = 1), and hxs = lhn e _ + o h(e). In continuous-time 
evolutions, whose realizations are continuous functions 
of time, the r dependence disappears from /i(e,r), so 
one can still define an e-entropy per unit time h(e). In 
particular, also in a pure deterministic flow one can put 
h(e) = h(e,T = 1). 

Let us remind that for a genuine deterministic chaotic 
system one has < h^s < °o {has = for a regular mo- 
tion), while for a continuous random process h^s — °°- 
Therefore, in order to distinguish between a purely deter- 
ministic system and a stochastic system it is necessary to 
perform the limit e — > in (|J). Obviously, from a phys- 
ical and/or numerical point of view this is impossible. 
Nevertheless, by looking at the behavior of the (e, r)- 
entropy at varying e one can have some qualitative and 
quantitative insights on the chaotic or stochastic nature 
of the process. For most of the usual stochastic processes 
one can explicitly give an estimate of the entropy scaling 
behavior when e ^ ||. For instance, in the case of a 
stationary Gaussian process one has p[ 
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h(e) = lim /i(e, r) ~ -s 



(5) 



Let us now introduce the main point of this letter by 
discussing in detail the difficulties that may arise in mea- 
suring the e-entropy for the following non-trivial example 
of a chaotic-diffusive map [pi , 



x t+i = x t + J» sin 27rx t 



(6) 



When p > 0.7326 . . ., this map produces a diffusive be- 
havior on large scales, so one expects 

h(e) ~ A for e < 1; h(e) oc ^ for e > 1, (7) 

where A is the Lyapunov exponent and D is the diffusion 
coefficient. The numerical computation of h(e), using 



the standard codification, is highly non-trivial already in 
this simple system. This can be seen by looking at Fig. 1 
where the behavior (Q) in the diffusive region is roughly 
obtained by considering the envelope of /i„(e,r) evalu- 
ated for different values of r; while looking at any single 
(small) value of r (one would like to put r = 1) one ob- 
tains a rather inconclusive result. This is due to the fact 
that one has to consider very large block lengths n when 
computing h(e, t), in order to obtain a good convergence 
for H n (e,T) — iJ„_i(e,r) in (^). Indeed, in the diffusive 
regime, a simple dimensional argument shows that the 
characteristic time of the system is T e « e 2 /D. If we 
consider for example, e = 10 and typical values of the 
diffusion coefficient D ~ 10 _1 , the characteristic time, 
T c , is much larger than the elementary sampling time 
t = 1. 




FIG. 1. Numerically evaluated (e, r)-entropy for the map 
([]) with p = 0.8 computed in the usual way [6] at r = 1 
(o), r = 10 (A) and r = 100 ( v) and different block length 
(n = 4, 8, 12, 20). The boxes (□) give the entropy computed 
with r = 1 by using periodic boundary condition over 40 cells. 
The latter is necessary in order to compute the Lyapunov ex- 
ponent A = has = 1.15. The straight lines correspond to the 
two asymptotic behaviors, h(e) = Iiks and h(e) ~ e -2 . 

Our approach to calculate h(e) differs from the usual 
one in the procedure to construct the coding sequence 
of the signal at a given level of accuracy. Specifically, 
we use a different way to sample the time, i.e., instead 
of using a constant time interval, r, we sample accord- 
ing to the exit-time, t(e), on an alternating grid of cell 
size e. We consider the original continuous-time record 
x{t) and a reference starting time t = to; the subsequent 
exit-time, t\, is then defined as the first time necessary 
to have an absolute variation equal to e/2 in x(t), i.e., 
\x(t(> + ti) — x(t )\ > e/2. This is the time the signal 
takes to exit the cell of size e. Then we restart from t\ to 
look for the next exit-time ti, i.e., the first time such that 
|o;(£o +ti — a?(*o | > e/2 and so on. Let us notice 
that, with this definition, the coarse-graining grid is not 
fixed, but it is always centered in the last exit position. 
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In this way we obtain a sequence of exit-times, {ti(e)}. 
To distinguish the direction of the exit (up or down out 
of a cell), we introduce the label ki — ±1, depending 
whether the signal is exiting above or below. Doing so, 
the trajectory is univocally coded with the required ac- 
curacy, by the sequence ((ti, ki), (t 2 , fe), • • • , (tM,k M )), 
where M is the total number of exit-time events observed 
during the total time T. Correspondingly, an "exit- 
time word" of length n is a sequence of couples of sym- 
bols £l"(e) = ((ti, h), (t i+1 , fcj+i), . . . , (ti +n -i, fe i+Tl _;i.)). 

From these words one firstly calculates the block en- 
tropies, Zf„(e), and then the exit-time e-entropies: 
h n (e) = limn-,00 H% +l (e) — H^(e). Let us notice that 
h n (e) is an e-entropy per exit and that M — T/{t(e)). 
The exit-time coding is a faithful reconstruction with ac- 
curacy e of the original signal. Therefore, the total en- 
tropy, Mh a (e), of the exit-time sequence, fi M (e), is equal 
to the total entropy, Th(e) — Nrh(e), of the standard 
codification sequence, W N (e). Namely, for the e-entropy 
per unit time, we obtain: 

ft(e)=M* n (e)/T=|M . ( 8 ) 

Now we are left with the determination of h (e). This 
implies a discretization, r e , of the exit-times. The exit- 
time entropy hP(e) becomes an exit-time (e, r e )-entropy, 
/i n (e, r e ), obtained from the sequence {r)i, ki}, where rji 
identifies the exit-time cell containing Equation (||) 
becomes now 

h(e) = lim h n (e,T e )/(t(e)) ~ h ( \e,r e )/ (t(e)) , (9) 

the latter relation being valid for small enough r e . How- 
ever, in all practical situations there exist a minimum r e 
given by the highest acquisition frequency, i.e., the limit 
r e — > cannot be reached. The discretization interval r e 
can be thought as the equivalent to the r entering in the 
usual (e, r)-entropy, so that 

h(e,T e ) = h n (e,T e )/(t(e)) . (10) 

At this point it is important to stress that in most of the 
cases the leading e contribution to h(e) in (^) is given 
by the mean exit-time (t(e)) and not by h (e, r e ). Any- 
how, the computation of h (e,T e ) is compulsory in or- 
der to recover a zero entropy for regular (e.g. periodic) 
signals. It is easy to obtain the following bounds for 
h n (e,T e ) = h n ({ Vi ,h}) : 

h n ({h})<h ( \{ m ,k t }) 

h n ({vuki}) < h Q ({ m })+h n ({h}) < h n ({h})+H?({ m }) 

where /i n ({fci}) is the Shannon entropy of the sequence 
{ki} and H^({r]i}) is the one-symbol entropy of the 
Therefore we have 



h n ({h}) <h( , < ^({fcj) + c( e )+ln((t( £ ))/r e ) 
{t{e)) ~ m - (t(e)) {U) 

where c(e) = — J p(z) hip(z)dz, and p(z) is the prob- 
ability distribution function of the rescaled exit-time 
z(e)=t(e)/(t(e)). 

We shall see in the following that the above bounds are 
rather good, and typically (t(e)) shows the same scaling 
behavior as h(e). One could wonder why the exit-time 
approach is better than the usual one. The reason is 
simple (and somehow deep): in the exit-time approach 
it is not necessary to use a very large block size since, 
at fixed e, the typical time at that scale is automatically 
given by (t(e)). This fact is particularly clear in the case 
of Brownian motion. In such a case (t(e)) oc e 2 /D, where 
D is the diffusion coefficient. As previously discussed, 
the computation of the h(e) with the standard methods 
implies the use of very large block sizes, of order e 2 /D. 
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FIG. 2. Numerically computed lower bound (□) and up- 
per bound (with r e = 1) (o) of h(e) ([H]) for the same map of 
Fig. 1. The two straight (solid) lines correspond to the asymp- 
totic behaviors as in Fig. 1. We also present the (e, r e )-entropy 
h n (e,r e )/(t(e)) with r e = 0.1(t(e)) (x). 

With our method the (t(e)) captures the correct scal- 
ing behavior and the exit-time entropy introduces, at 
worst, a sub-leading logarithmic contribution h^(e, r e ) ~ 
ln((t(e))/r e ). This is because c(e) is 0(1) and indepen- 
dent of e for a self-afnne signal and the h ({ki}) < ln2 
term is small compared with ln((t(e))/r e ) (for not too 
small e), so that, neglecting the logarithmic corrections, 
h(e) ~ l/(t(e)) oc De~ 2 . In order to clarify this point, 
we plot in Fig. 2 the calculation of the (e, r)-entropy via 
the exit-time approach for the previously discussed dif- 
fusive map. Fig. 2 has to be compared with Fig. 1 where 
the usual approach has been used. While in Fig. 1 the 
expected e-entropy scaling is roughly recovered as an en- 
velope over many different r, in our case the predicted 
behavior is easily recovered for any e with a remarkable 
improvement in the quality of the result. 
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Let us now briefly comment the limit e — * for a discrete- 
time system (e.g. maps). In this limit the exit-time ap- 
proach coincides with the usual one: we have just to ob- 
serve that practically the exit-times always coincide with 
the minimum sampling time and to consider the possi- 
bility to have jumps over more than one cell, i.e., the ki 
symbols may take values ±1, ±2, .... 




£ 

FIG. 3. Numerically computed lower bound (□) and upper 
bound (with r e = 0.1(t(e))) (o) for the (e, r e )-entropy in the 
case of a self-afnne signal with £ = 1/3 evaluated by using the 
exit-time approach. The two straight lines show the scaling 
e" 3 . 

As another example, we present the calculation of e- 
entropy for a self-afnne stochastic signal with Holder- 
exponent f = 1/3, i.e., \x{t) - x(t + At) | ~ (Ai) 1 / 3 . 
Such a signal can be seen as a stochastic surrogate of 
a turbulent signal (ignoring intermittency) and can be 
constructed in different ways (see Ref. JllJ and refer- 
ences therein). A simple dimensional estimate, which 
is rigorous for Gaussian processes tells us that the 
leading contribution to the e-entropy scaling is given by 
h(e) ~ e~ 3 . To generate the self-afhne signal we use a re- 
cently proposed algorithm |Tl|], where x(t) is obtained by 
using many Langevin processes. In Fig. 3 we show the 
bounds ( |rT| ) for (e, r e ) -entropy calculated via the exit- 
time approach. We observe an extended region of well- 
defined scaling, which is the same for l/(t(e)) ~ e~ 3 . The 
usual approach (not shown) gives a poor estimate for the 
scaling as the envelope of h(e, r) computed for various 
r (see for example Figs. 15-18 in |9]), as in the case of 
Fig. 1. 

In conclusion, we have introduced an efficient method to 
calculate the e-entropy based on the analysis of the exit- 
time statistics. It is able to disentangle in a more proper 



way the leading contributions to h(e) at the scale e, com- 
pared with the standard way based on a coarse-grained 
dynamics on a fixed (e, r) grid. We have presented the 
application to two examples, a chaotic diffusive map 
and a stochastic self-affine signal. More applications to 
chaotic systems, stochastic multi-affine processes, and ex- 
perimental turbulent signals will be reported elsewhere. 
The entropy-approach to evaluate the degree of com- 
plexity of a time series allows also to attack much more 
sophisticated problems often encountered in dynamical 
system theory. We just mention, e.g., the problem to 
disentangle correlations between time series obtained by 
measuring different observable of the same system. This 
might be addressed by using the conditional-entropy. 
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